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Abstract 

This  paper  documents  the  design,  development,  and  demonstration  of  prototype  automated 
performance  measures  for  staffs  using  advanced  command,  control,  communications,  computer, 
and  intelligence  (C4I)  systems.  These  prototype  measures  were  implemented  in  a  simulation- 
based  experiment  to  examine  the  impact  of  digital  systems  on  future  Battle  Command  training 
and  performance  assessment  at  the  battalion  and  brigade  level.  A  short  review  of  previous 
research  and  relevant  literature  on  automated  measures  is  presented,  followed  by  the  design  and 
development  of  the  prototype  measures,  and  a  discussion  of  sample  results  and  lessons  learned. 
More  detailed  information  can  be  found  in  Throne  et  al.  (1999).  The  complete  results  produced 
from  the  automated  measures  are  presented  in  the  five-volume  set  of  materials  entitled  Training 
and  Measurement  Support  Package,  Battle  Command  Reengineering  III,  Mounted  Maneuver 
Battlespace  Lab  (Training  and  Measurement  Support  Package,  1999).  Selected  results  from  this 
effort  are  also  included  in  the  Mounted  Maneuver  Battlespace  Lab’s  (1999a)  Experiment  Final 
Report. 


1  All  statements  expressed  in  this  paper  are  those  of  the  authors  and  do  not  necessarily  reflect  the  official  opinions  or  policies  of 
the  U.S.  Department  of  the  Army. 
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Introduction 


The  Army’s  growing  reliance  on  computer-mediated  work  (Caldera  &  Reimer,  1999) 
underscores  the  potential  of  digital  technologies  to  automatically  collect  and  analyze  real-world 
performance  data.  As  command,  control,  communications,  computer,  and  intelligence  (C4I) 
systems  become  more  integral  to  the  performance  of  individual  and  collective  tasks,  the  human- 
computer  interactions  associated  with  these  systems  become  more  critical  and  collectible. 
Digital  information  systems  potentially  can  provide  a  powerful  method  for  improving  staff 
training  and  performance  assessment  (Dwyer  et  al.,  1997). 

Researchers  involved  in  measuring  team  performance  (e.g.,  Cannon-Bowers  and  Salas,  1997) 
note  that  both  processes  and  outcomes  must  be  addressed.  Process  measures  address  the 
activities,  responses,  and  behaviors  that  people  use  to  accomplish  tasks.  They  include  planning, 
decision-making,  information  processing,  and  synchronization.  Many  process  measures  also 
have  corresponding  products,  such  as  decision  support  and  synchronization  matrices.  Efforts  to 
improve  staff  processes,  however,  must  also  be  linked  to  unit  outcomes,  such  as  execution 
activities  and  combat  effects  since  the  staff  functions  within  a  unit  and  its  mission  is  to  support 
command  and  control  (C~)  of  the  unit.  Outcome  measures  are  usually  not  diagnostic  by 
themselves  since  they  do  not  show  the  underlying  causes  of  performance,  or  how  to  improve  it. 
Thus,  it  is  necessary  to  collect  staff  process  and  product  measures  as  well  as  outcome  measures 
and  find  a  way  to  relate  them  to  each  other. 

Three  traditional  methods  of  performance  measurement  used  in  previous  performance 
assessment  research  are:  observation,  survey,  and  interview  (e.g.,  Campbell  et  al.,  1999).  These 
methods  have  traditionally  provided  the  measurement  basis  for  conventional,  that  is  pre-digital, 
staff  performance  assessment  (Crumley,  1989).  Together,  these  methods  can  yield  a 
multifaceted  look  at  staff  processes  during  training.  A  fourth  method,  examined  here,  is  the  use 
of  automated  measures  of  staff  performance. 

Digital  technologies  are  uniquely  suited  to  automatically  collect  user  performance  data.  In  fact, 
most  computers  can  and  do  routinely  log  or  track  all  user  inputs  and  system  responses.  Common 
examples  of  user  interactions  that  computers  routinely  log  include:  “Back”  keys,  lists  of  recently 
opened  files,  and  “Undo”  commands.  To  achieve  the  training  feedback  and  assessment  potential 
of  digital  technologies,  however,  computer  workstations  must  be  more  fully  instrumented 
(Lickteig  and  Throne,  1999).  For  clarification,  we  provide  a  working  definition  of  this 
instrumentation  requirement  for  computer-mediated  soldier  performance.  Instrumentation 
equates  to  a  log  of  all  soldier-computer  interactions  correlated  with  the  battlefield  situation  in 
which  they  occur.  Individual  logs  maintained  on  each  operator’s  C4I  device,  however,  are 
inadequate  for  examining  collective  performance  by  a  group  or  staff  of  individuals  working  as  a 
team.  Fortunately,  the  collaborative  nature  of  digital  media  readily  supports  a  network  of  C4I 
devices  and  an  integrated  log  of  soldier-computer  interactions  across  all  members  of  a  networked 
team.  In  fact,  this  digital  collaboration  extends  this  network  and  log  to  other  types  of  digital 
technologies  including  sensor  and  weapon  systems  as  well  as  simulation-based  exercises  in 
which  staff  performance  occurs. 


While  a  balanced  set  of  measurement  methods  is  often  required,  some  key  advantages  of 
automated  measures  are  briefly  considered.  First,  as  suggested,  the  more  workers  rely  on 
computers  to  do  their  work,  the  more  their  computer  interactions  will  become  meaningful  aspects 
of  work  processes  and  outcomes.  Second,  automated  measures  are  not  only  objective,  they  are 
direct  measures  of  performance.  In  contrast,  many  traditional  measurement  methods  are 
measures  about  performance.  Third,  a  greater  reliance  on  automated  measures  may  increase  the 
scope  and  precision  of  performance  assessment  and  feedback.  Fourth,  automated  measurement 
and  analysis  may  be  needed  for  more  complex  work  settings,  such  as  C~  staff  performance. 
Finally,  unobtrusive  and  automatic  data  collection  may  reduce  measurement  error  as  well  as 
observer  workload  and  resource  requirements. 

The  potential  of  automated  measures  was  demonstrated  during  the  Combat  Vehicle  Command 
and  Control  (CVCC)  program  conducted  by  ARI  at  Fort  Knox  (e.g.,  Leibrecht  et  al.,  1994).  The 
objective  of  the  CVCC  effort  was  to  determine  the  effect  of  vertically  integrating  digital  C" 
systems  in  an  armor  battalion.  To  assist  in  this  evaluation,  64  automated  measures  were 
developed.  These  measures  were  based  on  the  tactical  battlefield  operating  systems  for 
maneuver,  fire  support,  C  ,  and  intelligence.  Results  from  the  CVCC  program  of  research  used 
automated  measures  to  assess  training,  soldier-machine  interfaces,  and  operational  effectiveness. 

The  Army’s  growing  reliance  on  C4I  systems,  however,  underscores  the  need  for  additional 
research  and  development  on  automated  measures  based  on  the  ability  of  digital  technology  to 
support  training  assessment  and  feedback.  In  particular,  automated  measures  are  needed  that 
address  staff  performance  and  the  expanding  role  of  C"  systems  for  information-age  forces.  A 
key  challenge  is  how  to  obtain  automated  measures  of  staff  processes  and  products.  A 
subsequent  challenge  is  how  to  relate  these  automated  measures  to  unit  outcomes  in  order  to 
provide  more  effective  feedback  on  staff  performance. 

Project  Background 

In  response  to  the  concerns  and  issues  resulting  from  the  Army’s  digitization  efforts,  ARI’s 
research  unit  at  Fort  Knox  is  engaged  in  the  design  and  development  of  training  and  performance 
evaluation  techniques  to  support  Force  XXI  digital  capabilities  (ARI,  1997).  ARI’s  research  in 
this  area  seeks  to  capitalizes  on  recent  advances  in  the  cognitive  and  behavioral  sciences,  and  is 
focused  on  providing  an  empirical  foundation  for  improved  staff  training  and  evaluation 
strategies  for  the  digital  battlefield  of  the  future.  As  part  of  this  ARI  effort,  the  present  research 
project  entitled,  Prototype  Methods  for  the  Design  and  Evaluation  of  Training  and  Assessment  of 
Digital  Staffs  and  Crewmen  (DC4I)  was  performed  by  the  contractor  consortium  of  the  Human 
Resources  Research  Organization  (HumRRO),  Litton-PRC,  Klein  Associates,  and  Aptima. 

The  objective  of  the  DC4I  effort  was  to  design  prototype  training  and  evaluation  methods  for 
staff  operations  in  the  future  digital  tactical  operations  center  at  brigade  and  below.  The  research 
addressed  methods  for  training  leaders  and  staffs  of  future  digital  environments.  It  led  to  the 
development  of:  the  content  of  such  training,  the  methods  for  providing  the  training,  and  the 
means  for  assessing  performance  outcomes  as  a  result  of  training.  The  focus  of  this  paper  is  on 
the  last  issue,  the  assessment  of  digital  staff  processes  and  product  development  through 
automated  measures. 


An  opportunity  to  implement  the  automated  measures  of  staff  performance  developed  under 
DC4I  was  provided  by  the  Battle  Command  Reengineering  (BCR)  III  Concept  Experimentation 
Program  being  conducted  at  Fort  Knox  by  the  Mounted  Maneuver  Battlespace  Lab  (MMBL)  in 
the  Mounted  Warfare  Test  Bed  (MWTB).  The  DC4I  team,  in  reviewing  the  BCR  III  issues, 
determined  that  it  could  assist  in  answering  10  issues  based  on  the  team’s  military  staff  expertise. 
Six  issues  were  thought  to  be  potentially  answerable  through  automated  measures.  As  a  check 
on  the  automated  results,  traditional  measurement  methods  (survey,  observation,  and/or 
interview)  were  also  used  for  the  six  selected  measures  (Throne  et  al.,  1999).  A  complete  set  of 
the  BCR  III  evaluation  issues  and  methods  are  provided  in  the  MMBL’s  Experiment  Plan 
(MMBL,  1998).  Table  1  provides  the  subset  of  BCR  III  issues  focusing  on  staff  performance 
that  were  selected  for  measures  development. 

Table  1 

BCR  Issues  Addressed  by  PC4I  Measures 


Issues 

1.  Can  the  reengineered  Battle  Command  (RBC)  decrease  the  time  for  planning  and  increase 
the  time  to  prepare  and  rehearse?1'4 

2.  Can  the  RBC  provide  the  information  and  support  system  to  assist  the  Commander’s 
decision-making  process?1,2'3'4 

3.  Can  the  RBC  allow  efficient  synchronization  of  combat,  combat  support,  and  combat  service 
support  assets?1'3'4 

4.  Does  the  RBC  provide  efficient  battle  tracking  and  facilitate  precise  execution?1’3'4 

5.  Does  the  RBC  contribute  to  more  rapid  and  efficient  destruction  of  enemy  forces?1'4 

6.  Can  the  RBC  increase  the  span  of  control  of  the  Commander?1'3'4 

7.  What  effective  tactics,  techniques,  and  procedures  (TTPs)  are  emerging  from  the  RBC 
process?4 

8.  What  are  the  information  management  demands  on  multi-functional  officers  and  what  are  the 
impacts  of  these  demands  on  individual  and  team  performance,  process,  and  training 

2  3  4 

requirements?  ’  ' 

9.  What  are  the  impacts  of  RBC  on  team  performance  and  process  and  how  do  new  tools, 

2  4 

capabilities,  and  roles  change  the  task  force  organizational  architecture?" 

10.  What  are  the  impacts  of  RBC  on  perceived  and  actual  workload,  the  attribution  of  workload 
to  individual,  team,  and  task  demands,  and  individual  awareness  of  the  distribution  of 
workload  across  the  team?' 

Note.  1  =  addressed  by  automated  measures;  2  =  addressed  by  surveys;  3  =  addressed  by  observations;  4  = 
addressed  by  interviews. 

Specifications  for  the  evaluation  plan  began  with  analysis  of  the  issues  and  determination  of  the 
methods  that  could  best  be  used  to  address  each  issue.  The  data  collection  plan  was 
multifaceted,  involving  a  combination  of  automated  measures,  surveys,  observations,  and 


interviews.  The  specific  measures  (e.g.,  survey  and  interview  questions,  observation  points,  and 
automated  measures)  were  then  designed  and  developed  to  provide  broad  yet  efficient  coverage 
of  the  issues2.  These  research  issues  became  the  framework  around  which  the  automated 
measures  were  developed. 


Method 

Difficulties  associated  with  assessing  C2  team  performance  are  well  documented  and  include: 
evaluation  of  teams  vs.  individuals;  behaviors  that  are  difficult  to  observe  and  evaluate; 
complexity  of  environment;  and  no  standard  criteria  by  which  to  evaluate  performance  (Cooper 
et  al.,  1984;  Crumley,  1989).  These  are  valid  concerns  and  formidable  obstacles  for  which 
automated  measures  are  no  panacea.  However,  the  automated  measures  developed  during  this 
project  attempt  to  help  overcome  such  difficulties  and  extend  the  methods  available  for  assessing 
and  improving  staff  performance.  This  section  describes  how  automated  measures  of  staff 
performance  were  designed  and  developed. 

Analysis 

Measure  development  began  with  an  analysis  of  conventional  staff  performance  (Crumley, 
1989).  The  analysis  quickly  expanded,  however,  to  a  consideration  of  how  digital  systems  and 
future  staff  organizations  might  impact  staff  performance.  In  particular,  analysis  focused  on  how 
the  surrogate  command,  control,  communications,  and  computer  (SC4)  system  used  by  staff 
personnel  during  BCRs  impacted  staff  requirements  to  acquire  and  communicate  information. 

Staff  members  in  BCR  III  were  equipped  with  individual  SC4  systems  that  were  digitally 
networked  to  share  information  despite  being  geographically  separated  in  four  separate  C2 
vehicles,  as  depicted  in  Figure  1.  Each  soldier  on  the  staff  operated  an  SC4  workstation  that  was 
equipped  with:  an  electronic  map  that  displayed  automatic  updates  of  friendly  and  opposing 
forces’  (OPFOR)  location  and  status;  e-mail  messaging;  collaborative  mission  planning  tools, 
including  electronic  whiteboard  conferencing;  voice  communications;  and  a  computer-generated 
virtual  display  of  the  battlefield,  to  include  terrain  information  and  both  friendly  and  OPFOR 
vehicles  and  personnel. 

The  analysis  was  also  driven  by  the  fact  that  the  BCR’s  SC4  systems  were  designed  and 
developed  as  instrumented  systems.  Each  SC4  workstation  was  instrumented  to  acquire  data  on  a 
wide  range  of  soldier-SC4  system  interactions.  These  interactions  included  the  frequency  and 
duration  of  SC4  “tool”  use,  such  as  map  scaling  and  terrain  analysis,  as  well  as  information  flow 
based  on  SC4  communication  systems  such  as  e-mail  and  Whiteboards.  The  analysis  focused 
on  matching  BCR  research  issues  with  available  instrumentation  data.  Examples  of  the 
automated  measures  based  on  SC4  interactions  to  track  the  staff’s  use  of  digital  tools  and 
communication  systems  include  measures  2d  and  2e  in  Table  2. 


2 

"  Sample  measures  from  the  observations,  surveys,  and  interviews  can  be  viewed  in  Throne  et  al.  (1999)  or  the  complete  package 
can  be  viewed  in  the  Training  and  Measurement  Support  Package  (1999). 
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CONTROL  1 


Battle  Captain  (CPT) 
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Sensor  NCO  (SFC) 


COMMAND  1  (Command  Group) 
Commander  (LTC) 
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CONTROL  2 
Battle  Captain  (CPT) 
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Deputy  Commander  (MAJ) 

Ops  Officer  (CPT) 

OPS  NCO  (SFC) 


Figure  1.  Reengineered  Battalion  Command  Group  Structure. 


Measure  analysis  also  examined  how  the  mission  exercises  performed  by  the  staff  during  BCR 
III  were  conducted  in  the  Mounted  Warfare  Test  Bed  (MWTB)  using  distributed  interactive 
simulation.  This  virtual  simulation  environment  relies  on  a  distributed  networked  that  links  and 
records  all  manned  and  semi-automated  force  entities  including  sensors  and  weapon  system 
activation.  The  analysis  focused,  therefore,  on  matching  BCR  research  issues  with  available 
simulation  data.  Examples  of  the  automated  measures  based  on  simulation  data  to  track  the  use 
of  sensors  and  weapon  systems  include  measures  2a-2c  in  Table  2. 


Overall,  most  of  the  automated  measures  developed  and  listed  in  Table  2  exploit  the  availability 
of  both  SC4  interactions  and  simulation  data.  For  example,  many  of  these  staff  performance 
measures  assess  staff  interactions  with  SC4  systems  in  conjunction  with  simulation-based 
systems,  such  as  sensors  and  weapon  systems.  During  the  BCR’s  six  trials,  data  on  SC4 
interactions  and  simulation  events  were  automatically  logged  by  the  MWTB’s  Data  Collection 
Analysis  (DC A)  system.  The  temporally  synchronized  nature  of  the  DCA’s  composite  database 
supported  the  requirement  to  correlate  staff  process  and  product  development  to  exercise  events 
including  mission  outcomes.  The  measures  were  designed  with  these  capabilities  in  mind. 


Table  2 


Reengineered  Battle  Command  ( RBC)  Questions  and  Automated  Measures  of  Performance 


Question 

Measure  of  Performance 

Description 

1.  Can  the  RBC 
decrease  the  time  for 
planning  and  increase 
the  time  to  prepare 
and  rehearse? 

Battalion  staff 
communication  patterns 

Total  time  spent  using  each  of  the 
communication  tools  (e.g.,  e-mail, 
Whiteboard)  for  each  node  position  for 
each  planning  and  execution  session 

2.  Can  the  RBC 
provide  the 
information  and 

a.  UAV  mission 
effectiveness 

Number  of  opposing  force  vehicles  first 
detected  by  UAVs  divided  by  total 
number  of  OPFOR  vehicles  detected 

support  system  to 
assist  the 

Commander’s 

decision-making 

process? 

b.  Percent  of  enemy 
vehicles  inside  the 
battalion’s  area  of 
responsibility  that  were 
detected 

Number  of  unique  OPFOR  vehicles 
detected  by  sensors,  scouts,  or  weapons 
systems  controlled  by  the  battalion 
divided  by  the  number  of  OPFOR 
vehicles 

c.  Percent  of  enemy  vehicles 
inside  the  battalion’s  area  of 
responsibility  that  were 
never  detected 

Number  of  unique  OPFOR  vehicles  not 
detected  by  sensors,  scouts,  or  weapons 
systems  controlled  by  the  battalion 
divided  by  the  number  of  OPFOR 
vehicles 

d.  Use  of  SC4 

communication  tools  during 
mission 

Total  time  spent  using  each 
communication  tool  and  the  number  of 
communication  tool  initiations  per 
mission  for  each  node  position 

e.  Use  of  SC4  tools  allowing 
analysis  of  past,  present, 
and  projected  battlefield 
positions 

Total  time  spent  using  each  tool  and  the 
number  of  tool  initiations  for  each  node 

3.  Can  the  RBC  allow 
efficient 

synchronization  of 

a.  Percent  of  OPFOR  kills 
inside  effects  box 

Number  of  OPFOR  kills  in  effects  box 
divided  by  total  number  of  OPFOR 
kills 

combat,  combat 
support,  and  combat 
service  support 
assets? 

b.  Ratio  of  indirect  to  direct 
fire  OPFOR  kills 

Number  of  indirect  fire  kills  to  number 
of  direct  fire  kills 

( table  continues) 


3  The  unit’s  area  of  responsibility  was  delineated  by  its  rear,  flank,  and  forward  boundaries  assigned  by  its  higher  headquarters. 


Table  2  (Continued) 


Measure  of  Performance  Description 

a.  Percent  of  OPFOR  Total  number  of  flank  or  rear 

vehicles  engaged  from  flank  engagements  on  OPFOR  vehicles 
or  rear  divided  by  total  number  of  OPFOR 

vehicles 

b.  Percent  of  friendly  force  Total  number  of  flank  or  rear 

vehicles  engaged  from  flank  engagements  on  friendly  force  vehicles 
or  rear  divided  by  total  number  of  friendly 

force  vehicles 

c.  Average  range  of  OPFOR  Average  range  of  friendly  weapon 

fire  engagements  systems,  by  type,  against  OPFOR 

vehicles  that  were  killed  during  a 
mission 

a.  OPFOR  vehicle  kills  by  Number  of  OPFOR  vehicle  kills  by 

friendly  weapons  types  friendly  weapon  types  during  the 

mission 

b.  Time  to  destroy  OPFOR  Time  from  first  OPFOR  engagement 

until  OPFOR  vehicle  losses  exceed 

_ 70% _ 

6.  Can  the  RBC  Number  of  subordinate  unit  Commander’s  frequency  of  and  amount 

increase  the  span  of  leaders  Battalion  of  time  for  use  of  communications 

control  of  the  Commander  personally  tools  across  each  of  the  different 

Commander?  contacted  during  mission  personnel  with  which  he  interacted 

execution  during  mission  execution 

Design 

The  automated  measures  were  designed  to  address  the  six  MMBL  questions  listed  in  Table  2. 
Although  the  target  variables  in  these  research  questions  (e.g.,  efficient  synchronization,  span  of 
control)  were  not  completely  measurable  by  objective  data,  automated  measures  were  developed 
to  supplement  the  data  obtained  from  more  traditional  evaluation  methods  to  address  these 
issues. 

While  designing  the  automated  measures,  it  became  apparent  that  many  of  them  were  dependent 
upon  observer  assessment  of  when  certain  events  in  each  mission  began  or  ended.  Therefore,  an 
exercise  flag  log  was  developed  to  establish  the  data  collection  windows  for  each  exercise  (see 
Figure  3).  For  the  mission  events  listed  in  Figure  3,  only  a  few  could  be  independently  identified 
by  the  DCA  data  including  start  of  mission,  first  direct  fire  engagement,  and  the  end  of  mission. 
Start  and  stop  points  as  well  as  battlefield  locations  and  areas  for  the  rest  of  the  Table  2  events 
were  identified  by  observers  using  the  flag  log. 


5.  Does  the  RBC 
contribute  to  more 
rapid  and  efficient 
destruction  of  enemy 
forces? 


Question 

4.  Does  the  RBC 
provide  efficient 
battle  tracking  and 
facilitate  precise 
execution? 


Exercise  Flag  Log 

Today's  Date _  Exercise  ID 


Battalion  area  of  interest/responsibility 

Coordinates  of  upper  left  hand  corner  ( _ ) 

Coordinates  of  lower  left  hand  corner ( _ ) 

Coordinates  of  upper  right  hand  corner ( _ ) 

Coordinates  of  lower  right  hand  corner( _ ) 

Battalion  area  of  operation 

Coordinates  of  upper  left  hand  corner  ( _ ) 

Coordinates  of  lower  left  hand  corner( _ ) 

Coordinates  of  upper  right  hand  corner( _ ) 

Coordinates  of  lower  right  hand  corner( _ ) 

Start  of  Mission  (Time  [HH:MM:SS  ] _ : _ : _ ) 

Receipt  of  Brigade  order  (Time  of  Receipt  [HH:MM:SS  ] _ : _ : _ ) 

Start/end  of  planning 

(Start  time  of  mission  planning  [HH:MM:SS  ] _ : _ : _ ) 

(Stop  time  of  mission  planning  [HH:MM:SS  ] _ : _ : _ ) 

Start  of  mission  rehearsal  (Time  [HH:MM:SS  ] _ : _ : _ ) 

Start/end  of  execution  phase 

(Start  time  of  mission  execution  [HH:MM:SS  ] _ : _ : _ ) 

(Stop  time  of  mission  execution  [HH:MM:SS  ] _ : _ : _ ) 

Cross  line  of  departure  (Time  [HH:MM:SS  ] _ : _ : _ ) 

First  direct  fire  engagement  (  Time  [HH:MM:SS  ] _ : _ : _ ) 

Report  of  completion 

(Time  mission  completion  reported  [HH:MM:SS  ] _ : _ : _ ) 

Start  of  staff  consolidation 

(Start  time  of  staff  consolidation  [HH:MM:SS  ] _ : _ : _ ) 

End  of  mission  (Time  [HH:MM:SS  ] _ : _ : _ ) 


Figure  3.  Exercise  Flag  Log. 


Development 

An  initial  set  of  14  automated  measures  was  developed  for  the  BCR  III.  The  automated 
measures  focused  on  issues  regarding  the  effectiveness  and  efficiency  of  the  Reengineered  Battle 
Command  (RBC).  The  questions,  the  specific  measures  of  performance  designed  to  address 
them,  and  their  operational  definitions  can  be  seen  in  Table  2.  Although  these  measures  do  not 
tell  a  complete  story  about  staff  performance  on  their  own,  they  are  an  attempt  at  using 
automated  measures  to  assess  staff  processes  and  products  and  their  relation  to  unit  outcomes. 
Also,  they  become  more  informative  when  used  in  conjunction  with  more  traditional  process- 


oriented  measurement  methods,  such  as  observations  and  self-reports.  Although  beyond  the 
scope  of  this  paper,  examples  of  how  data  from  automated  and  traditional  measurement  methods 
were  used  to  address  BCR  issues  are  available  in  Throne  et  al.  (1999). 

The  BCR  III  data  analysis,  due  to  various  resource  limitations,  occurred  after  the  BCR  III  trials 
were  completed  and  the  training  unit  departed.  Once  the  recorded  experiment  exercise  files, 
which  had  a  complete  record  of  all  simulation  activities  and  individual  soldier-SC4  system 
interactions,  were  examined,  additional  data  collection  formats  had  to  be  designed  in  order  to 
pull  the  data  from  the  system  to  support  the  measures  of  performance  described  in  Table  2. 
Preliminary  radio  system  data  analyses  indicated,  for  example,  numerous  radio  transmissions  of 
1  second  or  less  in  length.  The  data  were  then  filtered  again  to  exclude  transmissions  of  this 
length  since  they  were  determined  to  contain  no  information  other  than  a  soldier  started  to 
transmit  a  message  but  stopped  before  any  information  could  be  conveyed. 

As  expected,  the  initial  DCA  outputs  on  automated  measures  included  numerous  other  data 
points,  so  considerable  data  examination,  editing,  and  subsequent  analyses  were  required.  For 
example,  when  trying  to  determine  the  number  of  OPFOR  combat  vehicles  the  experimental  unit 
fought  against  during  a  specific  mission,  off-line  filtering  of  the  DCA  data  was  required  to 
isolate  the  specific  vehicles  involved  from  additional  vehicles.  Most  of  these  additional  vehicles 
were  placed  into  the  exercise  to  set  the  conditions  for  the  following  experimental  trials. 

Notably,  comparability  of  results  was  not  an  analytic  consideration.  Performance  data  collection 
during  the  BCR  III  was  not  designed  to  evaluate  the  performance  of  the  unit  since  there  were  no 
doctrinal  or  baseline  standards  on  which  to  base  performance  measures  and  BCR  III  conditions 
did  not  permit  pre-  and  post-training  performance  measurements  of  the  unit.  However, 
comparability  is  an  important  issue  and  underscores  the  potential  for  change  in  staff  performance 
due  to  the  introduction  of  advanced  C4I  systems.  For  example,  C4I  tools  such  as  Whiteboards  for 
staff  conferencing  and  automated  terrain  intervisibility  checks  are  not  available  to  non  digital 
staffs.  While  analog  parallels  exist,  such  as  staff  huddles  and  map  exercises,  more  traditional 
measurement  methods  would  be  taxed  to  provide  the  precision  and  scope  about  staff  utilization 
that  is  readily  available  with  digital  tools  and  automated  measures.  More  fundamental  changes 
in  staff  performance  attributed  to  advanced  C4I  systems,  including  changes  to  the  Military 
Decision  Making  Process,  are  examined  in  the  MMBL’s  report  on  BCR  III  (MMBL,  1999a). 

Results 

To  illustrate  the  potential  of  automated  measures,  selected  results  pertaining  to  the  SC4  system’s 
capability  to  support  the  Commander’s  decision-making  process  are  provided4.  One  way  this 
was  measured  was  by  looking  at  the  performance  of  the  unmanned  aerial  vehicles  (UAVs), 
which  were  controlled  by  the  battalion  staff,  and  micro-UAVs,  which  were  controlled  by  the 
battalion  scout  platoon  (Table  2,  Measure  2a).  These  two  systems,  properly  employed,  could 
provide  the  Commander  with  accurate,  timely  information  about  the  location  and  activity  of  the 
OPFOR.  Over  the  course  of  the  nine  mission  trials  during  the  BCR  III,  the  UAVs  and  the  micro- 


4  A  complete  description  of  the  results  of  the  automated  measures  developed  for  BCR  III  can  be  found  in  the  Training  and 
Measurement  Support  Package  (1999). 


UAVs  were  the  first  to  detect  51%  of  all  OPFOR  vehicles  in  the  battalion’s  area  of  interest5.  In  a 
further  breakdown,  the  two  systems  detected  32%  of  tanks,  53%  of  infantry  fighting  vehicles 
(IFV),  and  56%  of  artillery  vehicles.  When  all  types  of  sensors  (i.e.,  ground,  airborne,  and 
satellite)  were  included  (Table  2,  Measure  2b),  70%  of  OPFOR  vehicles  were  detected. 

Notably,  the  majority  of  these  vehicles  were  first  detected  in  the  area  of  interest,  beyond  the  area 
of  operations,  which  gave  the  Commander  more  time  to  assess  OPFOR  strength,  capabilities,  and 
intentions  before  they  closed  to  within  range  of  the  unit’s  combat  systems.  For  the  30%  of 
OPFOR  vehicles  not  detected,  the  majority  were  second  echelon  tanks  or  artillery  systems  (Table 
2,  Measure  2c).  None  of  these  undetected  systems  had  an  impact  on  the  outcome  of  any  mission. 
Grouping  the  data  results  from  these  three  measures  together,  the  Commander  knew  the  location 
of  70%  of  the  OPFOR  vehicles  in  sufficient  time  to  make  an  informed  decision. 

Overall,  results  from  all  of  the  automated  measures  developed  under  this  project  helped  the 
MMBL  conclude  that  the  RBC  provided  the  information  and  support  system  needed  to  assist  the 
Commander’s  decision-making  process.  More  complete  results  for  the  sample  of  automated 
measures  described  as  well  the  other  automated  measures  listed  in  Table  2  are  available  in  the 
Mounted  Maneuver  Battlespace  Lab’s  (1999a)  Experiment  Final  Report. 

Conclusions 

Considerable  research  remains  to  be  done  on  developing  both  performance  standards  and 
evaluation  methods  for  future  battle  staffs  operating  advanced  C4I  systems.  While  surveys, 
observations,  and  interviews  were  used  to  gather  data  during  BCR  III,  the  project  team  focused 
on  developing  automated  measures  to  supplement  the  more  traditional  measurement  methods 
that  could  take  advantage  of  the  analytical  power  and  processing  speed  of  advanced  C4I  systems 
to  provide  real  or  near  real-time  feedback  to  the  training  participants.  Several  lessons  learned 
during  this  effort  concerning  automated  measures,  and  particularly  automated  measures  of  staff 
performance,  may  be  of  value  to  other  researchers. 

A  key  lesson  is  that  there  needs  to  be  extensive  collaboration  among  the  researchers,  subject 
matter  experts,  and  programmers  developing  automated  measures.  For  example,  operational 
definitions  may  need  to  be  revised  to  specify  the  format  in  which  the  automated  measures  data  is 
to  be  reported.  This  could  allow  the  programmer  extracting  data  to  set  up  the  output  so  it  can  be 
readily  converted  into  a  format  suitable  for  staff  performance  feedback  sessions.  The  DCA  used 
during  this  project  provides  the  programmer  with  a  multitude  of  methods  to  manipulate  and 
format  data.  Working  with  both  the  operational  definition  of  an  automated  measure  and  the 
specific  format  in  which  it  is  to  be  reported,  the  programmer  will  be  better  able  to  meet  the 
requirements  for  training  and  performance  assessment. 

A  second  related  lesson  is  that  measures  under  development  normally  require  iterative  runs  to 
refine  them.  The  initial  runs  may  provide  insights  into  reformatting  the  output  or  limiting  the 
parameters  to  exclude  extraneous  information.  Preliminary  analysis  of  the  data  could  also  point 
to  another  area  to  be  investigated  that  was  not  considered  during  the  analysis  and  design  of  the 
measures. 


5  The  battalion’s  area  of  interest  extended  15  kilometers  beyond  the  unit’s  area  of  responsibility. 


For  future  research,  standards  or  even  baseline  “case”  examples  of  staff  performance  would  be 
useful.  If  staff  processes  and  products  are  to  be  the  focus,  then  a  detailed  analysis  of  the 
processes  or  at  least  a  description  of  the  processes  would  help  guide  measures  development. 
This  analysis  should  include  who  is  involved,  what  tools  are  used,  and  what  products  are 
generated.  Even  if  unit  outcome  measures  will  not  be  the  primary  tool  to  provide  staff 
performance  feedback,  they  are  generally  easier  to  develop  than  staff  process  measures. 
Consequently,  they  should  always  be  considered  to  supplement  or  back  up  staff  process 
measures. 

Additional  research  on  and  development  of  automated  measures  of  staff  performance  was 
continued  for  the  BCR  IV  Experiment  at  Fort  Knox  during  April,  2000.  During  this  follow-on 
project,  DC4I-2,  the  research  team  attempted  to  refine  and  expand  the  automated  measures  that 
were  developed  for  BCR  III.  To  support  the  BCR  IV  effort,  a  draft  taxonomy  for  proposed 
automated  measures  was  introduced;  conceptual  formats  for  reporting  the  automated  measures 
were  developed;  and  a  team  consisting  of  programmers,  subject  matter  experts,  and 
psychologists  was  formed  to  guide  and  prioritize  the  development  effort.  The  results  of  this 
research  will  be  reported  as  they  become  available. 

In  summary,  while  the  work  reported  here  was  promising  and  useful  to  the  BCRs,  automated 
measurement  of  staff  performance  continues  to  have  unrealized  potential.  Additional  research  is 
required  to  determine  how  soldier-computer  interactions  can  be  used  to  measure  staff  processes 
and  products  rather  than  just  unit  outcomes  and  how  these  are  related.  As  C4I  systems  become 
more  integral  to  staff  performance,  automated  measures  should  help  achieve  more  balanced 
methods  for  measuring  staff  performance,  and  more  effective  and  immediate  feedback  for 
improving  staff  performance. 
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